Skip to content

chore(ci): fix coverage when modifying existing ported tests #2013

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 12, 2025

Conversation

spencer-tb
Copy link
Contributor

@spencer-tb spencer-tb commented Aug 8, 2025

🗒️ Description

The coverage CI doesn't handle cases where we modify existing ported tests correctly. This PR resolves that issue.

Problem

When modifying an existing test file that contains @pytest.mark.ported_from markers from past porting, the coverage CI would:

  • Detect existing ported markers in the modified file
  • Set any_ported=true
  • Skip generating BASE_TESTS for coverage comparison
  • Fail when trying to read non-existent BASE_TESTS index files

Solution

This fix operates on the principle that once a test is ported, the Python version becomes the source of truth. We distinguish between:

  • New porting operations: New @pytest.mark.ported_from markers in the git diff, means we only compare against original JSON from ethereum/tests.
  • Modifications to existing ported tests: No new @pytest.mark.ported_from markers in git diff, means we compare against the previous Python version.

Future Considerations

The coverage script is not capable of handling cases where new ported tests are added to an existing file that modifies existing tests. The recommendation for porting any new test from ethereum/tests is to a) add it to a new file but b) at a minimum do not modify any existing tests if added to an existing file.

🔗 Related Issues or PRs

#2002

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx --with=tox-uv tox -e lint,typecheck,spellcheck,markdownlint
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered adding an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).

@spencer-tb spencer-tb added scope:ci Scope: Continuous Integration type:chore Type: Chore labels Aug 8, 2025
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, I think we can make the approach a bit more consistent with the suggested changes, but let me know what you think.

Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I created a follow up issue to see if the other approach is worth implementing, meanwhile this should be ok.

@marioevz marioevz merged commit 54003b0 into main Aug 12, 2025
15 checks passed
@marioevz marioevz deleted the coverage-fix branch August 12, 2025 18:56
@spencer-tb
Copy link
Contributor Author

LGTM, I created a follow up issue to see if the other approach is worth implementing, meanwhile this should be ok.

Ahh thanks. Sorry couldn't get to it today!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope:ci Scope: Continuous Integration type:chore Type: Chore
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants